This chapter discusses the management of clusters. Clustering forms the basis of the SpinServer system's scaling and high-availability capacities. Every SpinServer in a system must be part of a cluster. A cluster can contain one or more SpinServers; however, a SpinServer can belong to only one cluster at a time.
This section discusses concepts and terms of cluster technology.
For ease of management of a SpinServer system and the data stored on it, you need to give some thought to the names you choose for the SpinServers in a cluster. To prevent confusion and potential loss or misplacement of data, the names you choose must be orderly and easily recognizable by all system administrators who maintain the SpinServer system.
Any naming system can be chosen for the system, but the IDs for the SpinServers must be unique within a cluster. After the ID for a SpinServer has been set, you cannot modify the server name or host name. To do so invalidates the failover port designation for the SpinServer and can result in the loss of data.
You enter the SpinServer ID into the system when you initially install the SpinFS software on the SpinServer. The name of the SpinServer is propagated from the initial installation into the management framework.
Initialization of a SpinServer involves joining an existing cluster or creating a new cluster. Every SpinServer is part of a cluster, and although the cluster can contain only a single SpinServer, each SpinServer must be entered into a cluster before any other system management task is undertaken.
Use the spinservers createcluster command to enter the first SpinServer into a cluster, and use the spinservers joincluster command to add a new SpinServer into an already existing cluster. The spinservers joincluster command provides all active SpinServers in the cluster with all of the communication details for contacting the newly configured SpinServer. The new SpinServer joins the cluster and quorum is achieved. (See Section 8.1.1 and Section 8.1.2 for information about quorum.) At that point, the new SpinServer is updated with the clusterwide view.
The spinserver createcluster and spinserver joincluster commands cause other system operations to take place. Both of the commands take the IP configuration for the cluster ports. This information is then stored in the interface configuration page after the node has completed the create or join cluster operation.
SpinServer clustering is based on the following concepts:
A quorum is formed when a majority of the known SpinServers, or nodes, in a cluster are healthy and in contact with one another. In this healthy state, they run a voting protocol that elects one node as the master; the remaining nodes are known as secondary nodes. (Note that the master/secondary designation in a cluster is unrelated to the concept of primary, active secondary, and passive secondary servers in a high-availability configuration.) When quorum is formed, it is maintained by continual voting of the nodes. Because of the majority requirement, there can be only one quorum at a time. Other nodes can join an existing quorum (for example, on restart). In a cluster's steady, healthy state, it is normal for a quorum to exist.
When a new quorum forms, a distributed recovery algorithm synchronizes all of the nodes. When this process successfully completes, all of the member nodes go online. When a new node joins an existing quorum, it individually proceeds through the recovery process and goes online.
In a steady state in a well-connected network, healthy nodes are online and each node is capable of running all defined SpinServer system operations. Changes made to one node are simultaneously propagated to all online nodes. When a node is offline (for instance, if there is no quorum, or if the node is unable to contact the quorum), it is not capable of running all defined SpinServer system operations.
In a cluster with an even number of nodes, epsilon enables a quorum to form by using slightly weaker requirements. Epsilon is an extra fractional voting weight held by, at most, one node. For example, two out of four eligible nodes are sufficient if one of those two nodes holds epsilon. Epsilon is a configuration setting for a node in a cluster.
An eligible node is one that has been designated by the administrator as an active participant in the cluster. Its presence impacts on the quorum membership requirements, and it is online (if at all possible). An administrator can choose to make the node ineligible for many reasons, such as recent software upgrades, network reconfiguration, and coping with failures. Use the cluster status enablenode and cluster status disablenode commands to configure node eligibility. If a node needs to be removed completely from the cluster, it must first be marked ineligible, and then removed from the cluster by using the cluster status disablenode command.
Managing clusters requires a working familiarity with the concepts that provide the framework for cluster technology in the SpinServer system. The primary concept that needs to be understood is the maintenance of quorum. Quorum is a precondition for a fully functioning cluster. When quorum is lost, the cluster loses the ability to accomplish normal cluster operations.
For single-node clusters, a majority is achieved when the single node is configured with epsilon. Achieving majority defines whether or not quorum is maintained; without epsilon, the majority is lost for the single-node cluster.
For this reason, a two-node cluster presents some unique challenges in maintaining quorum. In a two-node cluster, epsilon is not held by either node; instead, both nodes are continuously polled to ensure that if one node fails, the other has full read-write access to data as well as access to virtual interfaces and management functions.
In larger clusters, the effect of epsilon is most often felt when the cluster can be partitioned. In general, assuming reliable connectivity among the various nodes of the cluster, a larger cluster is more stable than a smaller cluster. The quorum requirement of a simple majority of half the nodes plus epsilon is easier to maintain in a cluster of 50 nodes than in a cluster of 2 nodes. However, when the connectivity between two equal portions of a large cluster fails, the group of SpinServers containing epsilon maintain quorum, assuming that all the SpinServers are healthy. For example, if a single link is established between 25 SpinServers in one room and 25 SpinServers in another room to comprise a 50-node cluster and the link fails, then the group of SpinServers that holds epsilon maintains quorum.
Only one collection of SpinServers can have quorum at any one time because all of the SpinServers collectively share a single view of the data. Therefore, if two noncommunicating SpinServers are permitted to modify the data in divergent ways, it is no longer possible to reconcile the data into a singular data view.
Managing clusters includes creating clusters, adding and removing SpinServers, and querying their status. Creating a cluster and adding SpinServers to it uses the spinservers createcluster and spinservers joincluster commands; the remaining tasks in the following list use the cluster command node.
Creating a cluster.
Adding additional SpinServers to the cluster.
Enabling or disabling a cluster node.
Adding or removing epsilon.
Enabling or disabling high-availability (HA) mode in a two-node cluster.
Displaying a cluster's configuration information.
Setting a route for a cluster.
Setting the audit-log size for a cluster.
Setting the cache size for a cluster.
Use the spinservers createcluster command to create a cluster from one SpinServer. The syntax of the command follows.
spinservers createcluster -ssid spinserver -clustername cluster_name -ipaddress1 IP_address [-ipaddress2 IP_address] -netmask1 netmask [-netmask2 netmask] [-broadcast1 IP_address] [-broadcast2 IP_address] [-mtu1 maximum_transmission_unit] [-mtu2 maximum_transmission_unit]
Important: This command must not be used for any purpose other than to create a new cluster. If you mistakenly run the spinservers createcluster command in attempt to add a SpinServer to an existing cluster, you must reinstall the SpinServer (or completely delete all configuration data) to undo the operation. Before you reinstall the SpinServer or delete any configuration information, contact Network Appliance Technical Support.
The following example initializes SpinServer Server1 into a cluster named Cluster1 with a cluster-network netmask of 255.255.0.0, an IP address of 192.168.66.1 for the first cluster interface, an IP address of 192.168.66.2 for the second cluster interface, a netmask of 255.255.0.0, a broadcast IP address of 192.168.255.255, and a maximum transmission unit (MTU) setting of 9000 bytes.
spinserver::> spinservers createcluster -ssid Server1 -clustername Cluster1 -ipaddress1 192.168.66.1 -ipaddress2 192.168.66.2 -netmask1 255.255.0.0 -broadcast1 192.168.255.255 -mtu1 9000
Note: The spinservers createcluster command can take up to 30 seconds to complete.
Use the spinservers joincluster command to join a SpinServer to a cluster. The syntax of the command follows.
spinservers joincluster -ssid spinserver -clusternodeipaddress IP_address -ipaddress1 IP_address [-ipaddress2 IP_address] -netmask1 netmask [-netmask2 netmask] [-broadcast1 IP_address] [-broadcast2 IP_address] [-mtu1 maximum_transmission_unit] [-mtu2 maximum_transmission_unit]
The following example adds a SpinServer named Server2 to a cluster that is hosted by a SpinServer with the IP address 192.168.66.10. The first cluster interface has the IP address 192.168.66.1, the second cluster interface has the IP address 192.168.66.2, the broadcast IP address is 192.168.255.255, the network mask is 255.255.0.0, and the MTU setting is 9000 bytes.
spinserver::> spinservers joincluster -ssid Server2 -clusternodeipaddress 192.168.66.10 -ipaddress1 192.168.66.1 -ipaddress2 192.168.66.2 -netmask1 255.255.0.0 -broadcast1 192.168.255.255 -mtu1 9000
Note: The spinservers joincluster command can take up to 30 seconds to complete.
After creating a cluster by using the spinservers createcluster command and adding other SpinServers to the cluster by using the spinservers joincluster command, you can perform the following cluster-management tasks:
Enabling and disabling quorum voting
Displaying information about cluster nodes
Adding epsilon to and removing epsilon from cluster nodes
Enabling and disabling two-node high-availability (HA) mode
Setting cluster routes
Setting the audit-log size for the cluster
To add or remove a SpinServer for quorum voting in a cluster, use the cluster status enablenode and cluster status disablenode commands. Enabling a SpinServer to participate in the quorum voting marks the node as eligible and changes the conditions under which quorum can be maintained. Quorum is maintained when half or more of the SpinServers that are eligible to vote are responding in a positive, healthy manner.
Note: The cluster status enablenode and cluster status disablenode commands cannot be run on the node that you are enabling or disabling. Run these commands on a node that is already a member of the cluster, and specify the node that is to be enabled or disabled by using the -ssid parameter.
Use the cluster status enablenode command to add a SpinServer to the quorum voting in a cluster. The command has the following syntax:
cluster status enablenode -ssid spinserver
The following example enables quorum voting for a SpinServer named Server1:
spinserver::> cluster status enablenode -ssid Server1
To remove a SpinServer from the quorum voting in a cluster and mark it as ineligible as an active participant in the cluster, use the cluster status disablenode command. The following example disables quorum voting for a SpinServer named Server1.
spinserver::> cluster status disablenode -ssid Server1
From the browser interface, the following confirmation message appears:
Disabling a node from the cluster means the node can no longer participate in cluster updates or voting. Are you sure you want to disable this node?
Click OK to confirm the command.
To add or remove the epsilon designation for a SpinServer in a cluster, use the cluster status addepsilon and cluster status removeepsilon commands. Epsilon enables a quorum to form by using slightly weaker requirements; for example, two out of four eligible nodes are sufficient if one of those two nodes holds epsilon. Quorum is maintained when half or more of the SpinServers that are eligible to vote are responding in a positive, healthy manner. A SpinServer can be designated as epsilon to add weight to its voting in a cluster with an even number of nodes. Only one SpinServer can be designated as epsilon at any given time in a cluster.
The cluster status addepsilon command is used to designate a SpinServer in a cluster as epsilon. The following example designates a SpinServer named Server1 as epsilon.
spinserver::> cluster status addepsilon -ssid Server1
The cluster status removeepsilon command is used to remove the designation of a SpinServer in a cluster as epsilon. The following example removes the designation of epsilon from a SpinServer named Server1.
spinserver::> cluster status removeepsilon -ssid Server1
To remove a node from a cluster, use the cluster status deletenode command. This command deletes all configuration data from the SpinServer. The command's -force parameter specifies whether the deletion operation is forced; the default is false. To return a deleted node to its original cluster or to add it to a different cluster, use the spinservers joincluster command.
The command has the following syntax:
cluster status deletenode -ssid spinserver -force {true | false}
The following example deletes a SpinServer named Server1 from its cluster. The deletion operation is not forced.
spinserver::> cluster status deletenode -ssid Server1 -force false
The cluster status show command displays the following information about nodes in a cluster:
Host name
Primary cluster IP address
Secondary cluster IP address
Epsilon designation
Node state
Health of node
The following example displays configuration information about the SpinServers in a cluster.
spinserver::> cluster status show
Hostname Cluster IP 1 Cluster IP 2 Epsilon Node State Health
Server1 10.3.128.20 10.3.128.28 true Eligible Online
Server2 10.3.128.21 10.3.128.27 false Eligible Online
You can set a specific network route for the cluster to control the flow of traffic across a complex network. Use the cluster routes commands to work with cluster routes.
Use the cluster routes new command to create a new cluster route. The command has the following syntax:
cluster routes new -portname port_name -nexthop next_hop_IP_address -netnumber network_IP_address -netmask netmask -metric metric
The following example creates a cluster route on port clus0 with a next-hop address of 127.127.0.0, a network IP address of 127.64.7.1, and a netmask of 255.255.0.0.
spinserver::> cluster routes new -portname clus0 -nexthop 127.127.0.0 -netnumber 127.64.7.1 -netmask 255.255.0.0 -metric 1
The cluster routes show command displays information about one or more cluster routes. The command has the following syntax:
cluster routes show [-portname port_name -nexthop next_hop_IP_address -netnumber network_IP_address -netmask netmask]
The following example displays information about all cluster routes:
spinserver::> cluster routes show Port Name Gateway Address Network Address Network Mask Metric clus0 127.127.0.0 127.64.7.1 255.255.0.0 1 clus1 127.127.0.0. 127.64.8.1 255.255.0.0 1
Use the cluster routes delete command to delete a cluster route. The command has the following syntax:
cluster routes delete -portname port_name -nexthop next_hop_IP_address -netnumber network_IP_address -netmask netmask
The following example deletes the route on the port clus0, which has a next-hop address of 127.127.0.0, a network IP address of 127.64.7.1, and a network mask of 255.255.0.0.
spinserver::> cluster routes delete -portname clus0 -nexthop 127.127.0.0 -netnumber 127.64.7.1 -netmask 255.255.0.0
High availability (HA) in a two-node cluster is a special case and must be specifically enabled by using the cluster configuration enablehamode command. If the cluster contains three or more nodes, you do not specifically need to enable HA mode in the cluster. See Basic Concepts of the SpinServer Architecture and Chapter 11 for more information on HA configurations.
Note: Before enabling HA mode, you must have a storage pool configured with an active secondary. See Section 11.2.15 and Section 11.2.17 for details.
To enable HA mode in a two-node cluster, use the cluster configuration enablehamode command. The command has the following syntax:
cluster configuration enablehamode -ssid spinserver
The following example enables HA mode in a two-node cluster that contains a SpinServer named Server1:
spinserver::> cluster configuration enablehamode -ssid Server1
Use the cluster configuration disablehamode command to disable HA mode in a two-node cluster. The command has the following syntax:
cluster configuration disablehamode -ssid spinserver
The following example disables HA mode in a two-node cluster that contains a SpinServer named Server1:
spinserver::> cluster configuration disablehamode -ssid Server1
The audit log stores a record of commands that are issued on a SpinServer cluster. By default, the number of commands recorded by the log is 10; commands are recorded across the entire cluster, not on individual SpinServers. The SpinServer's health-monitoring facility can be used to send periodic reports on the SpinServer's condition to an email address; this requires a mail-from address and outbound mail server (see Section 20.2.12 for details). If the cluster is predominantly serving either very large files or very small files, you can specify the use of a unified cache scheme to improve efficiency. You can specify whether or not a message is issued when a SpinServer in the cluster is restarted. Because restarts are categorized as critical, they often trigger pager alerts. You can set the number of commands that are contained in the audit log, the email properties, the cache scheme, and the sending of restart messages for a SpinServer cluster by using the cluster configuration modify command. The command has the following syntax:
cluster configuration modify -clustername cluster_name -auditlogsize audit_log_size -mailfrom mail_from_address -mailserver outbound_mail_server -unifiedcache {true | false} -sendmessage {true | false}
The following example changes the number of commands recorded in the audit log to 5000 in a cluster named cluster1. The mail-from address is set to postmaster@spincluster.example.com and the outbound mail server is specified as smtp1.example.com. The unified cache scheme is enabled, and restart messages are disabled.
spinserver::> cluster configuration modify -clustername cluster1 -auditlogsize 5000 -mailfrom postmaster@spincluster.example.com -mailserver smtp1.example.com -unifiedcache true -sendmessage false
Use the cluster configuration show command to display configuration information for a SpinServer cluster. The command has the following syntax:
cluster configuration show [-clustername cluster_name]
The following example displays configuration information for a cluster named cluster1:
spinserver::> cluster configuration show -clustername cluster1 Cluster Name cluster1 AuditLog Size 2000 Mail From postmaster@10.3.129.40 Outgoing SMTP Server foo Unified Cache Scheme Enabled true
Copyright © 2004 Network Appliance, Inc.; copyright © 2003, 2004 Spinnaker Networks, LLC. All rights reserved. Specifications subject to change without notice.